fix(language): make the parser setting real, and Chinese use jieba (#278) - #281
Conversation
HugoFara
added
enhancement
) A Chinese language created from the built-in preset produced a text with no clickable words at all: sentences=1, words=0. Nothing to look up, nothing to track. Three separate gaps stacked up to that. The parser setting was decorative. LgParserType was written by the language form and read by nothing — TextParsing::tokenize() branched only on the legacy MECAB magic word and otherwise went straight to StandardTextParser, so setting a language to jieba, mecab or even a bogus value parsed identically. The whole ParserRegistry existed only to populate the dropdown. It is now consulted before the built-in pipeline, and its tokens are adapted to ParsedToken; the two shapes differ only in sentence numbering and token ordering. External parsers never reached the dropdown either. ParserRegistry's constructor took an optional loader, and both direct call sites passed none, so registerExternalParsers() returned immediately: jieba was installed in the Docker image, working, and unlistable. It now builds a loader when given none. The presets could not express a parser. langdefs.json has carried parserType for the CJK languages all along, but LanguagePresets flattened it into an eight-slot tuple that dropped the field — which is why the Japanese preset, declaring mecab, silently resolved to character parsing. The tuple carries it as slot 8, the API exposes it, and both preset appliers set it. Chinese (Simplified) and (Traditional) now ask for jieba, keeping makeCharacterWord so the fallback stays meaningful. Opting in is deliberate and the fallback is safe. Only an explicit, non-default LgParserType routes to the registry; the legacy signals it also understands are ignored, so every language that exists today — all of which store no parser type — parses byte-identically. A parser the server cannot run falls back to the built-in pipeline, never to the regex parser, which for a CJK language yields zero words. The form gained an "Automatic" option so that "infer from the flags" stays expressible. Verified end to end against a real database with jieba installed: preset (jieba) LgParserType='jieba' sentences=2 words=7 no parser type LgParserType=NULL sentences=2 words=13 unavailable parser LgParserType='sudachi' sentences=2 words=13 where 7 is jieba word segmentation and 13 is one token per character.
fb0bf29 to
4c78c01
Compare
The opt-in test asked whether LgParserType was set, on the reasoning that no language stores one. They do. 20251223_120000_add_parser_type.sql backfills the column from the very legacy signals the opt-in test excludes: 'mecab' wherever the magic word sits in LgRegexpWordCharacters, 'character' wherever LgSplitEachChar is set. Every upgraded install with a CJK language therefore carries a parser type nobody chose, and routing it to the registry retokenizes the language. Measured with TextParsing::checkText() on a real 3.4.2 database, same texts, same settings, before and after: lg=2 Chinese (character, split=1) 12 sent / 103 words -> 13 / 122 lg=5 Japanese (character, split=1) 13 sent / 46 words -> 14 / 60 lg=1 French, lg=3 German, lg=4, 6, 7, 8 (no type) unchanged Existing texts keep their stored parse, so nothing breaks at upgrade; it diverges later and quietly, when UpdateLanguage re-parses on a settings change or a new text is imported. The language then holds texts split two different ways, and terms link by string, so saved vocabulary stops matching new occurrences. Read a type that only restates the flag beside it as the flag, not as a choice. 'character' with LgSplitEachChar, and 'mecab' with the magic word, carry no information the built-in pipeline is not already acting on. Anything else could only have been picked in the form: jieba, an external tokenizer, or 'character' on a language whose split flag is off, since the backfill never wrote that combination. Re-measured after the change: every language matches the pre-#281 baseline exactly. A language set to jieba still routes to jieba, and one naming a parser the server cannot run still falls back to the built-in pipeline. Deriving intent this way is a workaround for the magic word overloading LgRegexpWordCharacters, and can go once that is retired.
|
The safety claim in the description — "every language that exists today parses byte-identically; none of them stores a parser type" — doesn't hold on an upgraded install, and I caught it before merging by comparing parse output on a real 3.4.2 database.
UPDATE languages SET LgParserType = 'mecab'
WHERE UPPER(TRIM(LgRegexpWordCharacters)) = 'MECAB' AND LgParserType IS NULL;
UPDATE languages SET LgParserType = 'character'
WHERE LgSplitEachChar = 1 AND LgParserType IS NULL;So every character-split and MeCab language already carries an explicit value — exactly the condition that routes to the registry. Measured,
|
| Language | LgParserType |
before | with #281 as written |
|---|---|---|---|
| French (1) | (none) | 239 sent / 1058 words | identical |
| German (3) | (none) | 35 / 527 | identical |
| Chinese2 (4), Korean, Thai, Hebrew | (none) | — | identical |
| Chinese (2) | character |
12 / 103 | 13 / 122 |
| Japanese (5) | character |
13 / 46 | 14 / 60 |
The claim holds perfectly where no type is stored, and fails on precisely the languages the backfill touched.
Nothing breaks at upgrade — existing texts keep their stored parse, since reparseAllTexts() only runs on the legacy textitems cache path. It diverges later and quietly: UpdateLanguage already re-parses when LgParserType changes, and any new or edited text tokenizes the new way. The language then holds texts split two different ways, and since terms link by string, saved vocabulary stops matching new occurrences. That is a miserable bug to trace from a user report.
Fix pushed (17262f5)
A stored type that only restates the flag beside it is read as the flag, not as intent — character with LgSplitEachChar, mecab with the magic word. Both carry no information the built-in pipeline isn't already acting on. Anything else could only have been chosen in the form: jieba, an external tokenizer, or character on a language whose split flag is off, since the backfill never wrote that pair.
Re-measured after the change:
lg=1 French 239 / 1058 lg=2 Chinese 12 / 103 lg=5 Japanese 13 / 46
Every language back to the pre-#281 baseline, exactly. And the feature still works — on this install the registry lists regex, character, mecab, jieba, mecab-python all available:
lg=4 LgParserType |
result |
|---|---|
| (none) | 12 sent / 78 words |
jieba |
12 / 79 — the registry path runs |
sudachi-not-installed |
12 / 78 — falls back to the built-in pipeline, not to regex |
Psalm 0 errors · PHPCS clean under the project ruleset · ParserRegistryTest 20 pass, 4 new.
Follow-up, not for this PR
This is a workaround for the MECAB magic word overloading LgRegexpWordCharacters, which is what makes a stored value ambiguous in the first place. Filed separately — the magic word is still written by the language form today, and in SentenceService it means "no spaces between words" rather than "use MeCab", so retiring it is its own change with its own before/after comparison.
1 participant
Fixes #278.
The symptom, reproduced
A Chinese language created from the built-in preset parses a Chinese text to:
No clickable words at all — nothing to look up, nothing to track. That is the screenshot in the issue.
Three gaps stacked up to it
1. The parser setting was decorative.
LgParserTypewas written by the language form and read by nothing.TextParsing::tokenize()branches only on the legacyMECABmagic word and otherwise goes straight toStandardTextParser. Measured — the value makes no difference at all, including a value that isn't a parser:LgParserTypeNULLjiebaregexmecabtotally-bogusParserRegistryhad exactly two callers, both just filling the dropdown.CharacterParser,MecabParserandExternalParserwere unreachable from the parsing path.2. External parsers never reached the dropdown either.
ParserRegistry::__construct()takes an optional loader; both direct call sites passed none, soregisterExternalParsers()returned immediately. jieba is installed in the published Docker image (Dockerfile:59-65builds the venv and copiesparsers/, matchingconfig/parsers.php) and works — it was simply never listed.3. The presets could not express a parser.
langdefs.jsonhas carriedparserTypefor the CJK languages all along, butLanguagePresets::loadFromJson()flattened it into an eight-slot tuple that dropped the field. That is also why the Japanese preset, declaringmecab, silently resolved to character parsing even with MeCab installed.What changed
TextParsingconsults the registry before the built-in pipeline and adaptsToken→ParsedToken(the shapes differ only in sentence numbering and token ordering). The check-text preview uses the same parser as the save.ParserRegistrybuilds its ownExternalParserLoaderwhen given none, so jieba and MeCab Python appear on an install that has them.LanguagePresetscarriesparserTypeas slot 8; the API exposes it; both preset appliers (the Alpine store and the wizard's DOM applier) set it.makeCharacterWordso the fallback stays meaningful.Opting in is deliberate; the fallback is safe
Only an explicit, non-default
LgParserTyperoutes to the registry. The legacy signalsresolveParserTypeFromRow()also understands — theMECABmagic word andLgSplitEachChar— are deliberately ignored there, so every language that exists today parses byte-identically; none of them stores a parser type.A parser the server cannot run falls back to the built-in pipeline, never to the regex parser — which for a CJK language is precisely the zero-word text this issue is about.
Verified end to end, real database, jieba installed
7 is jieba word segmentation (
我 | 喜欢 | 学习 | 中文 | 这是 | 一个 | 句子); 13 is one token per character.Checks
Psalm 0 errors · PHPCS 0 errors, 0 warnings · PHPUnit 9101 pass (11 new, covering opt-in resolution, the legacy signals staying inert, the unavailable-parser fallback, and the preset slot) · Vitest 4326 pass · tsc and ESLint clean · assets rebuilt.
Not in this PR
/parse/route.services/nlp/has a second, complete jieba —JiebaParser, a live endpoint, and a PHP client inNlpServiceHandler::parse()/getAvailableParsers(). Both have zero callers. Two implementations, one a decoy; worth either wiring up or deleting.